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Introduction 

Staff in State Departments of Education are diligently reviewing and revising their state ac¬ 
countability systems to meet the new requirements and opportunities of the Every Student 
Succeeds Act (ESSA). ESSA is the latest reauthorization of the 1965 Elementary and Second¬ 
ary Education Act, the primary federal bill guiding K-12 education policy. As a democracy 
requires equity and adequate educational opportunities, this policy memo provides guid¬ 
ance to states for selecting more inclusive school quality and student success indicators for 
accountability systems. We do not prescribe a fixed list of indicators that each state should 
adopt, or even a fixed list of categories for states to consider. Rather, we use indicators of 
opportunities to learn and school climate to describe the risks and possibilities that states 
should consider when deciding on indicators. This memo concludes by recommending ap¬ 
proaches for selecting indicators that address the importance of equity and of students’ per¬ 
ceptions of support, safety, and respect in the classroom. 
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II. Overview of ESSA 


ESSA replaced No Child Left Behind, the previous version of the Elementary and Secondary 
Education Act (ESEA). While many significant changes are included, ESSA is still a primar¬ 
ily test-based accountability system, with states required to “identify a category of schools 
for comprehensive support and improvement” and then intervene after three years. The law 
specifies a set of academic indicators, each of which must be given “substantial weight” in 
the accountability system: 1 

1. Student proficiency on state assessments; 

2. English Language proficiency; and 

3. Another measure of academic student growth that can be disaggregated by sub¬ 
group (graduation rates for high schools; student growth on assessments or an¬ 
other measure for all other schools). 

4. At least one additional indicator of school quality or student success beyond test 
scores. 2 

The fourth indicator, the requirement that states include at least one indicator of school 
quality or student success, is the focus of this memo. While the choice of such so-called 
non-academic indicator(s) is left completely to the states, the law provides several examples 
of potential state-chosen measures or indicator(s): 

• student engagement; 

• educator engagement; 

• student access to and completion of advanced coursework; 

• postsecondary readiness; or 

• school climate and safety. 

As with the academic indicators, the non-academic indicator(s) must also be given “sub¬ 
stantial weight.” However, the law clearly requires that the accountability systems devised 
by states give “much” more weight, in the aggregate, to the academic indicators than to the 
non-academic indicator(s). (Note that from this point onward, we use just the singular “in¬ 
dicator”—but the reader should keep in mind that the law allows for either a single non-ac¬ 
ademic indicator or multiple indicators.) 

The requirement to include at least one non-academic indicator of school quality is new to 
federal policy, and it provides new opportunities for validly measuring broader purposes of 
schools. The new law requires that this indicator be valid, reliable, comparable, and state¬ 
wide. The indicator must also be able to distinguish among schools that differ with respect 
to performance on the indicator. That is, the indicator should show some schools doing well 
and some doing poorly. 

Collecting and analyzing data for a new non-academic indicator presents both a challenge 
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and an opportunity for state education leaders. The process of selecting an indicator that 
best aligns to the state’s accountability goals is not simple, nor is identifying a valid, reliable, 
comparable and statewide tool. Not surprisingly, research organizations, think tanks, advo¬ 
cates, and for-profit companies are bombarding state school leaders with topics, products 
and tools. 

Our goal in this policy memo is not to review the wisdom of this law or critique these prof¬ 
fered options, or even to endorse specific measures. Rather, our goal is to establish criteria 
for selecting non-academic indicators that state policymakers can use to maximize educa¬ 
tional equity and opportunity in their schools. 


III. Questions to Consider When Selecting Non-Academic Indicators 

Deciding what and how to measure change in students or a school is not easy. An indicator 
or collection of indicators provides everyone in the school system—students, parents, teach¬ 
ers, administrators, and community—with a signal about what is valued and where to focus 
their attention. NCLB focused attention on achievement test scores, specifically proficiency 
levels in English/ Language Arts and in mathematics. Years later, there is ample evidence 
that this limited focus created a perverse incentive to focus on some groups of students 
(“bubble kids”) over others 3 and to focus disproportionately on these two subject areas to 
the exclusion of others. 4 ESSA maintains NCLB’s tightly prescribed test-based emphasis. 
Yet, ESSA also gives states a limited new flexibility to broaden the goals for which schools 
are held accountable. Here are questions we suggest state and school leaders consider when 
making these important decisions. 


a. What types of indicators can help schools encourage equity? 

Under ESSA, states can choose indicators that shed light on the opportunity gaps that create 
and exacerbate achievement gaps, such as student access to higher-level coursework. 5 They 
can also, for example, gather data from schools on how different groups of students experi¬ 
ence their school environment. Choosing the right indicator for measuring the environment 
of the school can help focus attention on aspects of the education system that are critical to 
increasing equity and opportunity; e.g., improving safety for LGBTQ students 6 and students 
with disabilities, 7 and improving the racial climate experienced by different groups of stu¬ 
dents 8 and the level of trust among teachers and school leaders. 9 

Two categories of indicators are particularly valuable in increasing educational equity: stu¬ 
dent and teacher opportunity to learn and school climate— with a focus on teachers as well 
as students. There is strong research evidence that both types of indicators are positively 
linked to student outcomes, from achievement to social-emotional development and overall 
well-being. 10 

Opportunity to learn indicators encompass a broad range of possible measures. 11 For exam¬ 
ple, states may consider collecting information about the conditions in schools and districts 
that support teacher, parent, community, and student engagement. Such indicators would 
let us measure where and how schools are building the kinds of meaningful relationships 
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with families and the larger community that support student learning. Other types of oppor¬ 
tunity indicators includes evidence of preparation for future academic and career success, 
chances to contribute to civic life, and supportive conditions for scaling up effective pro¬ 
grams. 10 

Collecting opportunity to learn data across a state could provide state and district leaders 
with critical information about whether intended changes to schools and teaching are hap¬ 
pening. Such data help answer the question of whether all students are provided equitable 
and adequate opportunities to encounter curriculum and instruction that is of sufficient 
quality for students to meet state standards. For example, data on teacher opportunities 
to learn, whether from professional development or induction programs or as part of pro¬ 
fessional learning communities in schools, provide evidence of “educator engagement” in 
schools, one of the indicator categories included in ESSA. Though the category of opportu¬ 
nity to learn indicators is broad, it includes several specific measures that states could use 
to focus attention on aspects of the education system that research proves are critical to 
student and school success. 

School climate indicators provide information from the students’ perspective on their per¬ 
ceptions of safety, belonging, and the psychosocial impacts of the school community. Where¬ 
as suspension and expulsion data provide one data point that can track more extreme forms 
of student misbehavior and school discipline issues, school environment indicators such as 
surveys provide school leaders with broader and more precise data about the school climate. 
This can include who is more likely to experience bullying and harassment, who feels safe at 
school, and which groups of students do (or do not) have caring relationships with adults at 
school. Relationships with caring adults at school act as a protective factor for students and 
can improve student attendance, engagement, and academic performance. 12 

Additionally, how teachers and other staff experience the work environment of the school 
is a helpful related measure of school climate. Having a healthy and positive workplace that 
promotes collaboration, offers professional development opportunities, and fosters com¬ 
munity engagement all positively impact teacher retention in schools and student learn¬ 
ing. 13 Professional development that occurs in collaborative and collegial environments and 
provides authentic activities has been shown to generate improvements in student achieve¬ 
ment. 14 States and school districts should seek out measures that allow them to learn more 
about how students and teachers experience the climate of the school community. 


b. What are the potential risks of using these indicators? 

Not all indicators are valuable for promoting equitable school improvement. Some measures 
can be counterproductive as they may draw attention away from the conditions in schools 
that support or hinder student learning. A poorly considered measure might also end up 
unproductively and unfairly blaming school-level problems on individual students in those 
schools. There is, for instance, a strong risk of using measures of individual students’ psy¬ 
chological characteristics like grit 15 and persistence. Measures of grit that have been used in 
research are not ready for use in accountability systems. 16 

Selecting measures based on available evidence-based interventions carries some risks. On 
the one hand, ESSA demands that interventions be based on specific levels and types of ev- 
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idence. However, there are not enough interventions meeting the ESSA evidence-based cri¬ 
teria to solve the full range of educational problems educators and students face. For exam¬ 
ple, there are not yet interventions meeting ESSA’s evidenced-based threshold, that relate 
to creating safe and inclusive environments for LGBTQ students.. As an alternative, when 
there are no available evidence-based interventions that are useful to a particular school 
system, states and districts can build partnerships with researchers to develop, implement 
and measure new interventions. 17 


Some schools may be tempted to lean on systems they already have implemented. For in¬ 
stance, most schools already have some system for tracking and reporting school discipline 
data on violent crimes, suspensions, and expulsions. There are two main dangers in relying 
on these systems. First is the risk of choosing not to document, report, suspend, or expel 
students as an attempt at showing ‘improvement’ in their annual data. While policymakers 
may want to see these numbers decrease, this should be done as part of a deliberate and 
well-rounded effort to change a school’s approach to climate and discipline. As the NCLB 
experience demonstrated, chasing numbers does not necessarily create school improvement 
and can, in fact, be counterproductive. 


Given that a majority of 
the student-level variance 
in test scores is not 
attributable to schools, 
states must also take into 
consideration the reality 
that some indicators may 
be influenced by factors 
outside a school’s control. 


Second, these data only capture the most severe 
forms of violence and harmful behaviors. They also 
don’t provide information that would allow school 
leaders to more precisely diagnose which groups of 
students are most often targeted (and how and why 
this occurs). In addition, these indicators do not 
help identify the negative elements of school cli¬ 
mate that could be repaired through evidence-based 
intervention and prevention programs. 


Educator blaming is another potential pitfall for 
states as they look to adopt new indicators. In many 
accountability systems, states are attaching the per¬ 
formance of students on standardized tests to measures of teacher quality. While these ef¬ 
forts attempt to identify “effective” teachers and those who need more support, they gen¬ 
erate data solely about a narrow range of teacher impacts on students. This approach can 
erroneously blame individual educators rather than the systems that have created the op¬ 
portunity gaps for students in the first place. As compared with the currently popular trend 
of evaluating teachers on unstable student test scores, a much more productive approach is 
found in evaluating teachers on their use of standards for teacher practice that have been 
proven effective. These standards-based evaluation systems, such as the National Board 
Standards for teachers, have been found to produce student learning gains and are generally 
more supportive of teacher growth and development. 18 


Given that a majority of the student-level variance in test scores is not attributable to schools, 
states must also take into consideration the reality that some indicators may be influenced 
by factors outside a school’s control. Failure to account for these might hold a school respon¬ 
sible for something that is a result of district policy or of family and community factors (e.g., 
chronic absenteeism due to housing instability, employment insecurity, or limited availabil¬ 
ity of medical care). 


Finally, for any single indicator or bundle of indicators, there is great risk associated with 
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their integration into high-stakes accountability systems, because they can create perverse 
incentives to manipulate responses associated with accountability sanctions or rewards. Ex¬ 
cept for a few districts and states, 19 student success factors have not been used for account¬ 
ability purposes, so we do not know when, how, or why they might be manipulated. 


c. What evidence is there that a given measure can meaningfully and accu¬ 
rately distinguish schools of differing levels of quality and student success? 

As noted earlier, ESSA requires the non-academic indicator to vary significantly from school 
to school. Yet while analyses of student outcomes and perceptions of climate often reveal 
large individual differences, those differences can disappear when aggregated, resulting in 
relatively small differences between schools. Some research studies provide a comparative 
breakdown of how much variation is likely to be associated with a given indicator. In other 
words, they say how much schools tend to vary, as well as how much individual students 
or teachers vary on the measure. This information is important when selecting an indica¬ 
tor, so that the public can understand what a difference between schools actually means. 
If this guide post is not immediately clear, state leaders can always write to a member of 
the research team that developed the indicator to find out if there is evidence of signifi¬ 
cant school-level variation on the measure. Another problem can occur if a measure must 
discriminate among schools, but all schools across the state have high scores. In this case, 
“successful” schools might appear to be unsuccessful simply because the entire cohort is 
high scoring. This is an example of how choosing an indicator that meets ESSA’s criterion 
for discriminating among schools can create an unintended negative consequence, if stakes 
are attached to low scores. 


d. What measures of the indicator are available? And what validity and reli¬ 
ability evidence is there for measures of the indicator? 

As explained above, three key criteria for indicators are: 

• The measures should be able to meaningfully distinguish performance of schools, 
particularly with respect to their capacity to promote equity. 

• When an indicator is related to school or district policies and practices, account¬ 
ability for poor performance needs to be ascribed to entire schools or systems, rath¬ 
er than to individual educators or students. 

• The measures should take into consideration the range of factors both inside and 
outside a school’s control to change. 

To these, we add a fourth: 

• They must have demonstrably strong validity and reliability evidence. 

The validity of a given measure must be based on evidence from many studies, showing that 
it measures what is intended and can be used for the purposes intended. Reliability is an 
aspect of validity that refers in part to the stability of the measure across different adminis- 
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trations and sites where the measure is intended to be used. 

There are many indicators for which there is indeed strong evidence that they meaningful¬ 
ly distinguish among schools and are linked to differences in student outcomes. However, 
these measures were often validated in “no consequences” environments. Thus, high-stakes 
uses require new validity evidence. Hardly any measures, beyond tests, have been used for 
accountability purposes, so evidence on whether these indicators function as a worthy signal 
of what is important is limited. 

Below we share some examples of research-developed indicators—we list the indicator and 
provide a reference to research that supports it. This is by no means a comprehensive list of 
measures that states should consider, but rather an illustration of what some states, school 
districts and schools are working to develop. It is intended to support states’ search for valid 
indicators of measures of opportunities to learn and school climate that would be most use¬ 
ful to their unique context. 


Opportunities to Learn 

• Number of teachers who meet requirements for a regular teaching credential 20 

• Number of hours of extended learning time, such as through an extended school 
day, summer school, or afterschool learning 21 

• Student access to advanced coursework 22 

• Presenting intellectually challenging assignments to students 23 

• Instructional quality 24 

• Disproportionality in special education placements 25 

• Instructional coherence 26 


School Climate 

• Student and staff measures 

o Safety, relationships, norms & rules, teaching & learning 27 

• Staff measures 

o Trust among teachers 28 
o Teacher trust in principal 29 
o Collective efficacy 30 

o School working conditions (including teacher attrition and burnout) 31 
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Student measures 


o Connectedness to school 32 
o Chronic absenteeism 33 

o School facilities maintained and in good repair 34 
o Student safety, risky behaviors, and belonging 35 


IV. Recommendations 


As states and districts work to implement ESSA, non-academic indicators can signal a move 
away from punitive, high-stakes testing and towards a more holistic understanding of what 
helps students and educators thrive in school settings. Adapting measures that have a focus 
on equity and on improving school environments for the most vulnerable students can help 
states emphasize programs and interventions that can move schools in directions that pro¬ 
mote positive, healthy, and high-achieving environments. 

Yet deciding on and developing indicators that attend to equity is not a simple or quick 
process. It requires time to consider a state’s unique context—including its strengths and 
weaknesses in providing equitable education opportunities and safe school climates—along¬ 
side the potential to use or develop accurate indicators of progress. There is evidence that 
education interventions are effective when they are constantly being developed, tested and 
revised in a continuous cycle of inquiry. 36 Rather than treat the evolution of an account¬ 
ability system as a crisis, it is worth thinking about how to build change into the system as 
a constant. There are models around the country of local school systems that have existing 
democratic and research-driven approaches for developing and revising their accountability 
systems over time. 37 There are other examples of longstanding research and practice part¬ 
nerships in which school leaders and external researchers work together to create, measure 
and adapt education reform efforts over time. 38 

We suggest the following courses of action: 

• Identify indicators that signal the importance of equity, including opportunities to 
learn and/or creating safe and inclusive learning environments. 

• Adopt multiple non-academic indicators that states and schools can report in their 
annual report cards. States can do this even if they must, pursuant to ESSA, adopt 
a system with a single composite score. The design and presentation of this infor¬ 
mation can provide a far more comprehensive and authentic view of the schools to 
parents and the public. 

• Carefully combine indicators to signal what is important and avoid perverse incen¬ 
tives for manipulating any one indicator. 

• Create reciprocal accountability in which district and state leaders have respon¬ 
sibility to provide resources and create conditions needed to improve quality and 


http://nepc.colorado.edu/publication/ESSA 


8 of 12 



student success indicators. 


• Help schools make sense of data on quality and student success indicators by cou¬ 
pling them with opportunity and resource indicators. 

• Identify potential evidence-based resources ahead of time that can support schools 
in improving performance on the indicators (see What Works Clearinghouse for 
additional resources: http://ies.ed.gov/ncee/wwc/) and where there may be gaps in 
available resources to address issues surfaced by use of the indicators. 

• Develop an accountability plan that funds and supports school improvement for 
schools that need it, such as professional development and resources for identify¬ 
ing, adapting, and studying evidence-based programs. 

• Plan for a multi-stage rollout that can make new measurement approaches more 
successful and manageable over time. 
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